CV-Project

3D Gaussian Splatting is a technique used in computer vision and graphics for rendering 3D scenes. It represents 3D objects as Gaussian distributions in space, optimized via photometric loss. It has shown:

• High efficiency and real-time rendering capabilities.
• Advantages over neural network-based methods like NeRF due to simpler representation and rasterization compatibility.

Mip-Splatting Innovation

Mip-Splatting enhances 3D Gaussian Splatting by introducing:

1. 3D Smoothing Filters: To constrain the Gaussian primitives’ frequencies, ensuring adherence to sampling constraints and reducing high-frequency artifacts during zoom-in.

2. 2D Mip Filters: Aiming to mitigate aliasing when zooming out by simulating a box filter from physical imaging processes.

Slide 1: Introduction

Title: Mip-Splatting: Improvements for 3D Gaussian Splatting
Key Points:

• What is 3DGS?
  • Represents 3D scenes using Gaussian primitives (position, size, orientation).
  • Efficient real-time rendering but suffers from artifacts when scaling views.
• Problems in 3DGS:
  • High-frequency artifacts during zoom-in (thinning edges, erosion).
  • Brightness and dilation artifacts during zoom-out (over-smoothed or distorted details).
  • Poor generalization to new scales and resolutions.

---

Slide 2: Mip-Splatting Solutions

Title: How Mip-Splatting Solves 3DGS Issues

1. 3D Smoothing Filter

   • Applies low-pass filtering to Gaussian primitives in 3D space.
   • Constrains maximum frequencies to avoid high-frequency artifacts.
   • Effect: Eliminates thinning edges and erosion when zooming in.

2. 2D Mip Filter

   • Replaces 3DGS's dilation operation with a box-filter approximation.
   • Simulates the imaging process for anti-aliasing.
   • Effect: Prevents brightness/dilation artifacts during zoom-out.

---

Slide 3: Results and Impact

Title: Results of Mip-Splatting

1. Improved Rendering Quality

   • Consistent image fidelity across resolutions and scales.
   • Handles unseen sampling rates effectively (zoom-in/zoom-out scenarios).

2. Generalization Across Scales

   • Single-scale training achieves multi-scale rendering with no added overhead.

3. Efficiency

   • Dynamic Gaussian parameter adjustment reduces memory overhead.

Visuals:
Include one comparison figure (e.g., a zoom-in/zoom-out comparison showing reduced artifacts in Mip-Splatting).

---

When Change Resolution

Zoom in: 里的进的时候，辐条很细

Zoom Out: 离的远的时候，车轮膨胀

$$G_k(x{)}_{reg}=\sqrt{\frac{|{\mathrm{\Sigma }}_k|}{|{\mathrm{\Sigma }}_k+\frac{s}{v_k}\cdot I|}}exp(-\frac{1}{2}(x-p_k{)}^T({\mathrm{\Sigma }}_k+\frac{s}{v_k}\cdot I{)}^{-1}(x-p_k))$$

3D Smoothing Filter (Brief Overview)

Purpose:
Eliminates high-frequency artifacts (e.g., thinning edges) during zoom-in by constraining the frequencies of 3D Gaussian primitives.

---

Key Equation:
The smoothed Gaussian is defined as:

Where:

• ${\nu }_k$: Maximum sampling frequency of Gaussian $G_k$, computed from training views.
• $s$: Scalar controlling the filter size.
• $I$: Identity matrix.

---

Steps:

1. Compute ${\nu }_k$: Use Nyquist limit based on camera focal length and depth.
2. Apply Filter: Add a low-pass filter to the Gaussian covariance matrix.

---

Effect:
Prevents high-frequency artifacts, ensuring smooth rendering when zooming in on 3D scenes.

---

2D Mip Filter (Brief Overview)

Purpose:
Mitigates aliasing and brightness/dilation artifacts during zoom-out by simulating the physical imaging process with a box filter approximation.

---

Key Equation:
The filtered 2D Gaussian in screen space is defined as:

$$G_k^{2D}(x{)}_{mip}=\sqrt{(}\frac{{\mathrm{\Sigma }}_k^{2D}}{{\mathrm{\Sigma }}_k^{2D}+S\cdot I})exp(-\frac{1}{2}(x-p_k{)}^T({\mathrm{\Sigma }}_k^{2D}+S\cdot I{)}^{-1}(x-p_k))$$

Where:

• ${\mathrm{\Sigma }}_k^{2D}$: 2D covariance matrix of the projected Gaussian.
• $s$: Scalar controlling the filter size, chosen to match the pixel size.
• $I$: 2D identity matrix.

---

Steps:

1. Replace Dilation:
   Replace 3DGS’s screen-space dilation with this 2D Gaussian filter that approximates a box filter.

2. Adjust Filter Size:
   Set ss to cover a single pixel in screen space for consistent anti-aliasing.

---

Effect:

• Prevents brightness/dilation artifacts during zoom-out.
• Ensures smooth and natural rendering across scales with reduced aliasing.

Would you like more visual examples or implementation guidance?